Data Mining for the Corporate Masses?

F or several years, proponents have touted data mining as a powerful tool for finding patterns hidden in large databases. They promise many benefits, such as increased revenues for companies that use the technology to fine-tune their marketing by digging out customers' buying patterns from mountains of sales data. Until recently, however, data mining has been a complex, expensive, somewhat limited tool adopted primarily by large companies. This pattern may be changing, though, because of new techniques and technologies. Insurance companies and stock exchanges, for instance, are now using data mining to detect customer-activity patterns that indicate fraudulent behavior. And doctors are using data mining to predict the effectiveness of surgical procedures, tests, or medications for various types of conditions. Data mining, which automates the detection of complex patterns in databases , began formalizing as a technology in the mid-1990s. According to Dan Vesset, research director for market research firm IDC, the worldwide market grew from $455 million to $539 million last year, and is expected to continue increasing to $1.85 billion in 2006. Meta Group, a market research firm, the market leader is SAS Institute, followed by SPSS and IBM. The two most significant challenges driving changes in data mining are scal-ability and performance. Organizations want data mining to become more powerful so that they can analyze and compare multiple data sets, not just individual large data sets, as is traditionally the case. They also want to break up data into finer-grained categories for analysis. Scalability is critical because databases have become very large. Terabyte-class databases have become more common today, particularly as the cost of storage has decreased and the amount of e-commerce has increased. IBM, for example, claims more than 200 customers with data warehouses larger than a terabyte, several of which are more than 30 ter-abytes. And industry observers predict that within a few years, some users will have 100-terabyte data warehouses. As companies capture more data, managing and mining the information become more complex. In some cases, data miners have tried to solve this problem by analyzing small samples of large data sets. However, said Dan Graham, IBM's director of business-intelligence solutions , this approach is a concession to the limitations of today's data mining hardware and software. Data mining yields better results if more data is analyzed , he explained. The growth in e-commerce is driving the need for data mining approaches that work …